Added public incident report for Degradation in email, SMS, and realtime delivery #20
Conversation
WalkthroughAdds Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests
Tip 👮 Agentic pre-merge checks are now available in preview!Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.
Please see the documentation for more information. Example: reviews:
pre_merge_checks:
custom_checks:
- name: "Undocumented Breaking Changes"
mode: "warning"
instructions: |
Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).Please share your feedback with us on this Discord post. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
2025/08/24/readme.md (5)
3-5: Confirm timestamps and metadataPlease verify start/end times against the status page and consider adding Severity and Regions to the header block for quick scanning.
9-9: Define OTP on first mention and standardize casing of “Realtime”Improves clarity and consistency.
-Between 24–26 August 2025, some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email OTPs, and realtime events. The issue was caused by instability in our messaging system. The problem was resolved by stabilizing the messaging infrastructure, and all services are now fully operational. +Between 24–26 August 2025, some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email one-time passwords (OTPs), and Realtime events. The issue was caused by instability in our messaging system. The problem was resolved by stabilizing the messaging infrastructure, and all services are now fully operational.
19-21: Format affected components as a bulleted listCurrent lines read as run-on text; list improves scanability.
-Email delivery (invitations, password resets, notifications) -OTP delivery via email and SMS -Realtime events +- Email delivery (invitations, password resets, notifications) +- OTP delivery via email and SMS +- Realtime events
64-64: Wording: singular “overload”Tiny consistency tweak.
-Improve monitoring and alerting for earlier detection of overloads. +Improve monitoring and alerting for earlier detection of overload.
62-65: Action items: add owners and due datesHelps drive accountability. Example:
-### Action items - -Improve monitoring and alerting for earlier detection of overload. +### Action items + +- Improve monitoring and alerting for earlier detection of overload. (Owner: SRE On-call; Due: 2025-09-15) +- Add queue depth/error-rate SLOs with paging alerts. (Owner: Platform Eng; Due: 2025-09-20)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
2025/08/24/readme.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
2025/08/24/readme.md
[grammar] ~3-~3: There might be a mistake here.
Context: ...Incident Start: 2025-08-24 07:26 UTC - Incident End: 2025-08-26 05:10 UTC - *...
(QB_NEW_EN)
[grammar] ~4-~4: There might be a mistake here.
Context: ...- Incident End: 2025-08-26 05:10 UTC - Report Prepared By: Divyansha ## Summ...
(QB_NEW_EN)
[grammar] ~19-~19: There might be a mistake here.
Context: ...tations, password resets, notifications) OTP delivery via email and SMS Realtime ...
(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ... analysis ### Preliminary findings We noticed few errors in the messaging system resp...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
2025/08/24/readme.md
70-70: Bare URL used
(MD034, no-bare-urls)
|
|
||
| ### Preliminary findings | ||
|
|
||
| We noticed few errors in the messaging system responsible for handling message delivery. |
There was a problem hiding this comment.
Grammar: “a few errors”
Minor clarity fix.
-We noticed few errors in the messaging system responsible for handling message delivery.
+We noticed a few errors in the messaging system responsible for handling message delivery.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| We noticed few errors in the messaging system responsible for handling message delivery. | |
| We noticed a few errors in the messaging system responsible for handling message delivery. |
🧰 Tools
🪛 LanguageTool
[grammar] ~31-~31: There might be a mistake here.
Context: ... analysis ### Preliminary findings We noticed few errors in the messaging system resp...
(QB_NEW_EN)
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 31, the phrase "We noticed few errors in the
messaging system responsible for handling message delivery." uses incorrect
grammar; change "few" to "a few" so the sentence reads "We noticed a few errors
in the messaging system responsible for handling message delivery." to improve
clarity and correctness.
|
|
||
| ### Resolution | ||
|
|
||
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload. |
There was a problem hiding this comment.
Grammar: pluralize “task” and add article
Also reads better with “the overload”.
-We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload.
+We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling tasks contributing to the overload.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload. | |
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling tasks contributing to the overload. |
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 49, the sentence uses the singular “task”
and omits an article before “overload”; change it to pluralize “task” to “tasks”
and add “the” so the phrase reads, for example, “disabling the tasks
contributing to the overload,” resulting in: “We stabilized the messaging
infrastructure by migrating critical queues to a more reliable system and
disabling the tasks contributing to the overload.”
|
|
||
| ### Supporting documentation | ||
|
|
||
| https://status.appwrite.online/incident/711585 No newline at end of file |
There was a problem hiding this comment.
Fix bare URL (MD034) with descriptive markdown link
Resolves markdownlint warning.
-https://status.appwrite.online/incident/711585
+[Status incident 711585](https://status.appwrite.online/incident/711585)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| https://status.appwrite.online/incident/711585 | |
| [Status incident 711585](https://status.appwrite.online/incident/711585) |
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
70-70: Bare URL used
(MD034, no-bare-urls)
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 70, replace the bare URL
"https://status.appwrite.online/incident/711585" with a descriptive markdown
link to resolve MD034; update it to use bracketed link text like "[Appwrite
status: Incident 711585]" (or another concise description) followed by the URL
in parentheses so the markdown is more descriptive and passes markdownlint.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (4)
2025/08/24/readme.md (4)
1-1: Use a descriptive, user-facing incident title with datesImproves indexing and clarity.
-# Email, SMS, and Realtime Delivery Issues +# Degradation: Email, SMS, and Realtime Delivery (Aug 24–26, 2025)
31-31: Grammar: “a few errors”-We noticed few errors in the messaging system responsible for handling message delivery. +We noticed a few errors in the messaging system responsible for handling message delivery.
49-49: Grammar: pluralize “task” and add article-We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload. +We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling tasks contributing to the overload.
70-70: Fix bare URL with descriptive markdown link (MD034)Resolves markdownlint warning.
-https://status.appwrite.online/incident/711585 +[Status incident 711585](https://status.appwrite.online/incident/711585)
🧹 Nitpick comments (6)
2025/08/24/readme.md (6)
3-6: Add duration and severity to top metadataGives quick at-a-glance context.
- **Incident Start:** 2025-08-24 07:26 UTC - **Incident End:** 2025-08-26 05:10 UTC +- **Duration:** 45h 44m +- **Severity:** Degraded - **Report Prepared By:** DivyanshaPlease verify the duration math from the provided timestamps and adjust severity per your taxonomy.
9-10: Clarify timezone in summary date rangeKeeps consistency with UTC metadata.
-Between 24–26 August 2025, some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email OTPs, and realtime events. +Between 24–26 August 2025 (UTC), some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email OTPs, and realtime events.
15-16: Mention alerting and time to detectAdd whether automated alerts fired and the TTD metric to strengthen learnings.
25-26: Quantify impact where possibleInclude approximate error rate, affected regions/tenants, or % of messages delayed/failed.
39-40: Clarify the “why” behind the overloadBriefly note precipitating factors (traffic spike, config/regression, dependency limits) and what prevented backpressure. Helps readers understand recurrence risk.
62-64: Make action items SMART (owner, due date, measurable outcome)Improves accountability and follow-through.
-Improve monitoring and alerting for earlier detection of overloads. + - [ ] Add queue depth and consumer lag SLOs with alerts (Owner: @oncall-backend, Due: 2025-09-30) + - [ ] Implement rate limiting/backpressure on producers to protect queues (Owner: @platform, Due: 2025-10-07) + - [ ] Load test messaging path to 2× peak with dashboards (Owner: @perf, Due: 2025-10-14; Success: <1% drop rate at 2×)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
2025/08/24/readme.md(1 hunks)
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
2025/08/24/readme.md
70-70: Bare URL used
(MD034, no-bare-urls)
What does this PR do?
Added public incident report for Degradation in email, SMS, and realtime delivery on 2025-08-24
Test Plan
(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your changes work.)
Related PRs and Issues
(If this PR is related to any other PR or resolves any issue or related to any issue link all related PR and issues here.)
Have you read the Contributing Guidelines on issues?
(Write your answer here.)
Summary by CodeRabbit